-
Notifications
You must be signed in to change notification settings - Fork 39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cron job maintenance #448
Cron job maintenance #448
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I love the idea of providing overridable cronjob templates with unlimited cron jobs. Given however, that the default cron jobs are all run daily, what if we just have a single bash script that is invoked by a single cronjob, which will in turn invoke the 3 cleanup commands sequentially? This can be overridden or replaced as desired.
I was going to suggest the same thing, but I wanted to ensure the syntax for defining multiple cron jobs was working correctly first. There is already a maintenance.sh script we could use as-is, or modify for our needs. |
Makes sense to switch to the maintenance script if we can. |
…urations to be parameterized.
Please ensure #453 has been merged before merging this. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me! Can this be taken out of draft?
eb6d704
to
64320c8
Compare
Co-authored-by: Nuwan Goonasekera <2070605+nuwang@users.noreply.github.com>
|
||
To disable a cron job after Galaxy has been deployed simply set the schedule to a date that | ||
can never occur such as midnight on Februrary 30th: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Better to add an enabled flag as we have for everything else?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I considered that, but in other cases when enabled: false
is set we don't render the template. However, in this case it means the CronJob
is not defined and not available to be run manually. By setting the schedule to a time that never occurs the CronJob
is defined and is available to be run manually if desired. At least that was my thought process.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reading more about cron jobs, maybe I should be using .spec.suspend
here. I'll play around with that and see if I get the desired behavior.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good idea to investigate suspend. Maybe we should add both. Some instructions on how to manually invoke the cronjob would also be useful.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Using .spec.suspend
looks to be less than ideal, particularly if one wants to run the job once but keep it suspended. In that case the user must:
- Patch the cron job to set
.spec.suspend=false
- Run the job if it does not get triggered automatically, and
- Patch the cron job again to re-set
.spec.suspend=true
Given users can always disable a cron job with the schedule
I am not sure .spec.suspend
is worth it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that adding an enabled flag would still be useful. Users who want to run the cronjob manually can still use the trick you suggested. Just that it feels weird to have to think of an invalid date for the simplest case of not wanting the cron job.
Co-authored-by: Nuwan Goonasekera <2070605+nuwang@users.noreply.github.com>
Co-authored-by: Nuwan Goonasekera <2070605+nuwang@users.noreply.github.com>
4adc64d
to
07fdbd4
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ksuderman Looks like we have a conflict, and I've made some more minor comments, but other than that, this looks like it's good to go?
|
||
To disable a cron job after Galaxy has been deployed simply set the schedule to a date that | ||
can never occur such as midnight on Februrary 30th: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think that adding an enabled flag would still be useful. Users who want to run the cronjob manually can still use the trick you suggested. Just that it feels weird to have to think of an invalid date for the simplest case of not wanting the cron job.
Adds cron jobs that clean up the
jobs_directory
, datasets, and histories. The jobs are defined in thevalues.yaml
file and templates generated from thecronjob-maintenance.yaml
template. The cron jobs are based on the Ansible Playbook used on mainSince the
/galaxy/server/database/tmp
directory is cleaned up I am not sure the job that cleans thetus_upload_store
is needed since it also lives in/galaxy/server/database/tmp
, but I've included it for completeness.The
pgcleanup.py
script writes logs to thetmp
directory, but we should likely create a separate logs directory for these and then also prune the logs as needed.Should we just call the
maintenance.sh
script, or a modified version of the script?NOTE I have set the cron jobs to
runAsUser: 0
but we should probably run these as the Galaxy user once we have determined whymutliqc
is creating files and directories asnobody
.This is untested as of yet.
Closes #408